New Approaches to Temporal Abstraction in Hierarchical Reinforcement Learning

In classical reinforcement learning, planning is done at the level of atomic actions, which is highly laborious for complex tasks. By using temporal abstraction, an agent can construct plans more e ciently through considering di erent levels of detail. This thesis investigates new approaches to automatically discover and represent temporal abstractions. Two methods are introduced: succession statistics and proximity statistics. Both collect statistics on state pairs. Succession statistics capture the behavior of the agent during training, which allows to make predictions about future trajectories for any pair of start and goal states. Proximity statistics describe how close any two states are. This information can be used to perform macro value updates across distant states. Through ignoring large parts of all possible state pairs, the statistics can be made sparse. To obtain a good compromise between the bene t of the statistics and their size, important bottlenecks in the state space, which are called gate points, are automatically identi ed and used for sparsi cation. Experiments with statistics on state pairs show their potential to make planning more e cient. For an example problem, the number of states that have to be considered during planning decreases to 16% if the full succession statistics are used compared to uninformed search. Using a sparse variant, which only considers 4% of all state pairs, the method still reduces the steps during search to 35%. Information from the succession statistics can also be utilized by prioritized sweeping such that it improves by 25-30% over its uninformed variant. Finally, the number of updates for value iteration can be reduced to only 12% when the normal micro updates are interleaved with macro updates using the proximity statistics. This thesis does not solve the problem of temporal abstraction, but it provides a new perspective to it. It is related to existing approaches, especially to the options framework, and methods from other elds, like the successor state representation or transit node routing.